{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summarizing Text with Amazon Reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据集：Amazon 500000评论\n",
    "\n",
    "\n",
    "本节内容:\n",
    "- 数据预处理\n",
    "- 构建Seq2Seq模型\n",
    "- 训练网络\n",
    "- 测试效果\n",
    "\n",
    "seq2seq教程: https://github.com/j-min/tf_tutorial_plus/tree/master/RNN_seq2seq/contrib_seq2seq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow Version: 1.1.0\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "import re\n",
    "from nltk.corpus import stopwords\n",
    "import time\n",
    "from tensorflow.python.layers.core import Dense\n",
    "from tensorflow.python.ops.rnn_cell_impl import _zero_state_tensors\n",
    "print('TensorFlow Version: {}'.format(tf.__version__))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insepcting the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "reviews = pd.read_csv(\"Reviews.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(568454, 10)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>ProductId</th>\n",
       "      <th>UserId</th>\n",
       "      <th>ProfileName</th>\n",
       "      <th>HelpfulnessNumerator</th>\n",
       "      <th>HelpfulnessDenominator</th>\n",
       "      <th>Score</th>\n",
       "      <th>Time</th>\n",
       "      <th>Summary</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>B001E4KFG0</td>\n",
       "      <td>A3SGXH7AUHU8GW</td>\n",
       "      <td>delmartian</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>1303862400</td>\n",
       "      <td>Good Quality Dog Food</td>\n",
       "      <td>I have bought several of the Vitality canned d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>B00813GRG4</td>\n",
       "      <td>A1D87F6ZCVE5NK</td>\n",
       "      <td>dll pa</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1346976000</td>\n",
       "      <td>Not as Advertised</td>\n",
       "      <td>Product arrived labeled as Jumbo Salted Peanut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>B000LQOCH0</td>\n",
       "      <td>ABXLMWJIXXAIN</td>\n",
       "      <td>Natalia Corres \"Natalia Corres\"</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1219017600</td>\n",
       "      <td>\"Delight\" says it all</td>\n",
       "      <td>This is a confection that has been around a fe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>B000UA0QIQ</td>\n",
       "      <td>A395BORC6FGVXV</td>\n",
       "      <td>Karl</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1307923200</td>\n",
       "      <td>Cough Medicine</td>\n",
       "      <td>If you are looking for the secret ingredient i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>B006K2ZZ7K</td>\n",
       "      <td>A1UQRSCLF8GW1T</td>\n",
       "      <td>Michael D. Bigham \"M. Wassir\"</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "      <td>1350777600</td>\n",
       "      <td>Great taffy</td>\n",
       "      <td>Great taffy at a great price.  There was a wid...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id   ProductId          UserId                      ProfileName  \\\n",
       "0   1  B001E4KFG0  A3SGXH7AUHU8GW                       delmartian   \n",
       "1   2  B00813GRG4  A1D87F6ZCVE5NK                           dll pa   \n",
       "2   3  B000LQOCH0   ABXLMWJIXXAIN  Natalia Corres \"Natalia Corres\"   \n",
       "3   4  B000UA0QIQ  A395BORC6FGVXV                             Karl   \n",
       "4   5  B006K2ZZ7K  A1UQRSCLF8GW1T    Michael D. Bigham \"M. Wassir\"   \n",
       "\n",
       "   HelpfulnessNumerator  HelpfulnessDenominator  Score        Time  \\\n",
       "0                     1                       1      5  1303862400   \n",
       "1                     0                       0      1  1346976000   \n",
       "2                     1                       1      4  1219017600   \n",
       "3                     3                       3      2  1307923200   \n",
       "4                     0                       0      5  1350777600   \n",
       "\n",
       "                 Summary                                               Text  \n",
       "0  Good Quality Dog Food  I have bought several of the Vitality canned d...  \n",
       "1      Not as Advertised  Product arrived labeled as Jumbo Salted Peanut...  \n",
       "2  \"Delight\" says it all  This is a confection that has been around a fe...  \n",
       "3         Cough Medicine  If you are looking for the secret ingredient i...  \n",
       "4            Great taffy  Great taffy at a great price.  There was a wid...  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Id                         0\n",
       "ProductId                  0\n",
       "UserId                     0\n",
       "ProfileName               16\n",
       "HelpfulnessNumerator       0\n",
       "HelpfulnessDenominator     0\n",
       "Score                      0\n",
       "Time                       0\n",
       "Summary                   26\n",
       "Text                       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check for any nulls values\n",
    "reviews.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Remove null values and unneeded features\n",
    "reviews = reviews.dropna()\n",
    "reviews = reviews.drop(['Id','ProductId','UserId','ProfileName','HelpfulnessNumerator','HelpfulnessDenominator',\n",
    "                        'Score','Time'], 1)\n",
    "reviews = reviews.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Summary</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Good Quality Dog Food</td>\n",
       "      <td>I have bought several of the Vitality canned d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Not as Advertised</td>\n",
       "      <td>Product arrived labeled as Jumbo Salted Peanut...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Delight\" says it all</td>\n",
       "      <td>This is a confection that has been around a fe...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Cough Medicine</td>\n",
       "      <td>If you are looking for the secret ingredient i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Great taffy</td>\n",
       "      <td>Great taffy at a great price.  There was a wid...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 Summary                                               Text\n",
       "0  Good Quality Dog Food  I have bought several of the Vitality canned d...\n",
       "1      Not as Advertised  Product arrived labeled as Jumbo Salted Peanut...\n",
       "2  \"Delight\" says it all  This is a confection that has been around a fe...\n",
       "3         Cough Medicine  If you are looking for the secret ingredient i...\n",
       "4            Great taffy  Great taffy at a great price.  There was a wid..."
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reviews.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review # 1\n",
      "Good Quality Dog Food\n",
      "I have bought several of the Vitality canned dog food products and have found them all to be of good quality. The product looks more like a stew than a processed meat and it smells better. My Labrador is finicky and she appreciates this product better than  most.\n",
      "\n",
      "Review # 2\n",
      "Not as Advertised\n",
      "Product arrived labeled as Jumbo Salted Peanuts...the peanuts were actually small sized unsalted. Not sure if this was an error or if the vendor intended to represent the product as \"Jumbo\".\n",
      "\n",
      "Review # 3\n",
      "\"Delight\" says it all\n",
      "This is a confection that has been around a few centuries.  It is a light, pillowy citrus gelatin with nuts - in this case Filberts. And it is cut into tiny squares and then liberally coated with powdered sugar.  And it is a tiny mouthful of heaven.  Not too chewy, and very flavorful.  I highly recommend this yummy treat.  If you are familiar with the story of C.S. Lewis' \"The Lion, The Witch, and The Wardrobe\" - this is the treat that seduces Edmund into selling out his Brother and Sisters to the Witch.\n",
      "\n",
      "Review # 4\n",
      "Cough Medicine\n",
      "If you are looking for the secret ingredient in Robitussin I believe I have found it.  I got this in addition to the Root Beer Extract I ordered (which was good) and made some cherry soda.  The flavor is very medicinal.\n",
      "\n",
      "Review # 5\n",
      "Great taffy\n",
      "Great taffy at a great price.  There was a wide assortment of yummy taffy.  Delivery was very quick.  If your a taffy lover, this is a deal.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Inspecting some of the reviews\n",
    "for i in range(5):\n",
    "    print(\"Review #\",i+1)\n",
    "    print(reviews.Summary[i])\n",
    "    print(reviews.Text[i])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据预处理\n",
    "\n",
    "- 全部转换成小写\n",
    "- 连词转换\n",
    "- 去停用词（只在描述中去掉）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "contractions = { \n",
    "\"ain't\": \"am not\",\n",
    "\"aren't\": \"are not\",\n",
    "\"can't\": \"cannot\",\n",
    "\"can't've\": \"cannot have\",\n",
    "\"'cause\": \"because\",\n",
    "\"could've\": \"could have\",\n",
    "\"couldn't\": \"could not\",\n",
    "\"couldn't've\": \"could not have\",\n",
    "\"didn't\": \"did not\",\n",
    "\"doesn't\": \"does not\",\n",
    "\"don't\": \"do not\",\n",
    "\"hadn't\": \"had not\",\n",
    "\"hadn't've\": \"had not have\",\n",
    "\"hasn't\": \"has not\",\n",
    "\"haven't\": \"have not\",\n",
    "\"he'd\": \"he would\",\n",
    "\"he'd've\": \"he would have\",\n",
    "\"he'll\": \"he will\",\n",
    "\"he's\": \"he is\",\n",
    "\"how'd\": \"how did\",\n",
    "\"how'll\": \"how will\",\n",
    "\"how's\": \"how is\",\n",
    "\"i'd\": \"i would\",\n",
    "\"i'll\": \"i will\",\n",
    "\"i'm\": \"i am\",\n",
    "\"i've\": \"i have\",\n",
    "\"isn't\": \"is not\",\n",
    "\"it'd\": \"it would\",\n",
    "\"it'll\": \"it will\",\n",
    "\"it's\": \"it is\",\n",
    "\"let's\": \"let us\",\n",
    "\"ma'am\": \"madam\",\n",
    "\"mayn't\": \"may not\",\n",
    "\"might've\": \"might have\",\n",
    "\"mightn't\": \"might not\",\n",
    "\"must've\": \"must have\",\n",
    "\"mustn't\": \"must not\",\n",
    "\"needn't\": \"need not\",\n",
    "\"oughtn't\": \"ought not\",\n",
    "\"shan't\": \"shall not\",\n",
    "\"sha'n't\": \"shall not\",\n",
    "\"she'd\": \"she would\",\n",
    "\"she'll\": \"she will\",\n",
    "\"she's\": \"she is\",\n",
    "\"should've\": \"should have\",\n",
    "\"shouldn't\": \"should not\",\n",
    "\"that'd\": \"that would\",\n",
    "\"that's\": \"that is\",\n",
    "\"there'd\": \"there had\",\n",
    "\"there's\": \"there is\",\n",
    "\"they'd\": \"they would\",\n",
    "\"they'll\": \"they will\",\n",
    "\"they're\": \"they are\",\n",
    "\"they've\": \"they have\",\n",
    "\"wasn't\": \"was not\",\n",
    "\"we'd\": \"we would\",\n",
    "\"we'll\": \"we will\",\n",
    "\"we're\": \"we are\",\n",
    "\"we've\": \"we have\",\n",
    "\"weren't\": \"were not\",\n",
    "\"what'll\": \"what will\",\n",
    "\"what're\": \"what are\",\n",
    "\"what's\": \"what is\",\n",
    "\"what've\": \"what have\",\n",
    "\"where'd\": \"where did\",\n",
    "\"where's\": \"where is\",\n",
    "\"who'll\": \"who will\",\n",
    "\"who's\": \"who is\",\n",
    "\"won't\": \"will not\",\n",
    "\"wouldn't\": \"would not\",\n",
    "\"you'd\": \"you would\",\n",
    "\"you'll\": \"you will\",\n",
    "\"you're\": \"you are\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def clean_text(text, remove_stopwords = True):\n",
    "    '''Remove unwanted characters, stopwords, and format the text to create fewer nulls word embeddings'''\n",
    "    \n",
    "    # Convert words to lower case\n",
    "    text = text.lower()\n",
    "    \n",
    "    # Replace contractions with their longer forms \n",
    "    if True:\n",
    "        text = text.split()\n",
    "        new_text = []\n",
    "        for word in text:\n",
    "            if word in contractions:\n",
    "                new_text.append(contractions[word])\n",
    "            else:\n",
    "                new_text.append(word)\n",
    "        text = \" \".join(new_text)\n",
    "    \n",
    "    # Format words and remove unwanted characters\n",
    "    text = re.sub(r'https?:\\/\\/.*[\\r\\n]*', '', text, flags=re.MULTILINE)\n",
    "    text = re.sub(r'\\<a href', ' ', text)\n",
    "    text = re.sub(r'&amp;', '', text) \n",
    "    text = re.sub(r'[_\"\\-;%()|+&=*%.,!?:#$@\\[\\]/]', ' ', text)\n",
    "    text = re.sub(r'<br />', ' ', text)\n",
    "    text = re.sub(r'\\'', ' ', text)\n",
    "    \n",
    "    # Optionally, remove stop words\n",
    "    if remove_stopwords:\n",
    "        text = text.split()\n",
    "        stops = set(stopwords.words(\"english\"))\n",
    "        text = [w for w in text if not w in stops]\n",
    "        text = \" \".join(text)\n",
    "\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will remove the stopwords from the texts because they do not provide much use for training our model. However, we will keep them for our summaries so that they sound more like natural phrases. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summaries are complete.\n",
      "Texts are complete.\n"
     ]
    }
   ],
   "source": [
    "# Clean the summaries and texts\n",
    "clean_summaries = []\n",
    "for summary in reviews.Summary:\n",
    "    clean_summaries.append(clean_text(summary, remove_stopwords=False))\n",
    "print(\"Summaries are complete.\")\n",
    "\n",
    "clean_texts = []\n",
    "for text in reviews.Text:\n",
    "    clean_texts.append(clean_text(text))\n",
    "print(\"Texts are complete.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clean Review # 1\n",
      "good quality dog food\n",
      "bought several vitality canned dog food products found good quality product looks like stew processed meat smells better labrador finicky appreciates product better\n",
      "\n",
      "Clean Review # 2\n",
      "not as advertised\n",
      "product arrived labeled jumbo salted peanuts peanuts actually small sized unsalted sure error vendor intended represent product jumbo\n",
      "\n",
      "Clean Review # 3\n",
      " delight  says it all\n",
      "confection around centuries light pillowy citrus gelatin nuts case filberts cut tiny squares liberally coated powdered sugar tiny mouthful heaven chewy flavorful highly recommend yummy treat familiar story c lewis lion witch wardrobe treat seduces edmund selling brother sisters witch\n",
      "\n",
      "Clean Review # 4\n",
      "cough medicine\n",
      "looking secret ingredient robitussin believe found got addition root beer extract ordered good made cherry soda flavor medicinal\n",
      "\n",
      "Clean Review # 5\n",
      "great taffy\n",
      "great taffy great price wide assortment yummy taffy delivery quick taffy lover deal\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Inspect the cleaned summaries and texts to ensure they have been cleaned well\n",
    "for i in range(5):\n",
    "    print(\"Clean Review #\",i+1)\n",
    "    print(clean_summaries[i])\n",
    "    print(clean_texts[i])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def count_words(count_dict, text):\n",
    "    '''Count the number of occurrences of each word in a set of text'''\n",
    "    for sentence in text:\n",
    "        for word in sentence.split():\n",
    "            if word not in count_dict:\n",
    "                count_dict[word] = 1\n",
    "            else:\n",
    "                count_dict[word] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of Vocabulary: 132884\n"
     ]
    }
   ],
   "source": [
    "# Find the number of times each word was used and the size of the vocabulary\n",
    "word_counts = {}\n",
    "\n",
    "count_words(word_counts, clean_summaries)\n",
    "count_words(word_counts, clean_texts)\n",
    "            \n",
    "print(\"Size of Vocabulary:\", len(word_counts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用构建好的词向量"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word embeddings: 484557\n"
     ]
    }
   ],
   "source": [
    "# Load Conceptnet Numberbatch's (CN) embeddings, similar to GloVe, but probably better \n",
    "# (https://github.com/commonsense/conceptnet-numberbatch)\n",
    "embeddings_index = {}\n",
    "with open('numberbatch-en-17.04b.txt', encoding='utf-8') as f:\n",
    "    for line in f:\n",
    "        values = line.split(' ')\n",
    "        word = values[0]\n",
    "        embedding = np.asarray(values[1:], dtype='float32')\n",
    "        embeddings_index[word] = embedding\n",
    "\n",
    "print('Word embeddings:', len(embeddings_index))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of words missing from CN: 3044\n",
      "Percent of words that are missing from vocabulary: 2.29%\n"
     ]
    }
   ],
   "source": [
    "# Find the number of words that are missing from CN, and are used more than our threshold.\n",
    "missing_words = 0\n",
    "threshold = 20\n",
    "\n",
    "for word, count in word_counts.items():\n",
    "    if count > threshold:\n",
    "        if word not in embeddings_index:\n",
    "            missing_words += 1\n",
    "            \n",
    "missing_ratio = round(missing_words/len(word_counts),4)*100\n",
    "            \n",
    "print(\"Number of words missing from CN:\", missing_words)\n",
    "print(\"Percent of words that are missing from vocabulary: {}%\".format(missing_ratio))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "阈值设置为20，不在词向量中的且出现超过20次，那咱们就得自己做它的映射向量了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of unique words: 132884\n",
      "Number of words we will use: 65469\n",
      "Percent of words we will use: 49.27%\n"
     ]
    }
   ],
   "source": [
    "# Limit the vocab that we will use to words that appear ≥ threshold or are in GloVe\n",
    "\n",
    "#dictionary to convert words to integers\n",
    "vocab_to_int = {} \n",
    "\n",
    "value = 0\n",
    "for word, count in word_counts.items():\n",
    "    if count >= threshold or word in embeddings_index:\n",
    "        vocab_to_int[word] = value\n",
    "        value += 1\n",
    "\n",
    "# Special tokens that will be added to our vocab\n",
    "codes = [\"<UNK>\",\"<PAD>\",\"<EOS>\",\"<GO>\"]   \n",
    "\n",
    "# Add codes to vocab\n",
    "for code in codes:\n",
    "    vocab_to_int[code] = len(vocab_to_int)\n",
    "\n",
    "# Dictionary to convert integers to words\n",
    "int_to_vocab = {}\n",
    "for word, value in vocab_to_int.items():\n",
    "    int_to_vocab[value] = word\n",
    "\n",
    "usage_ratio = round(len(vocab_to_int) / len(word_counts),4)*100\n",
    "\n",
    "print(\"Total number of unique words:\", len(word_counts))\n",
    "print(\"Number of words we will use:\", len(vocab_to_int))\n",
    "print(\"Percent of words we will use: {}%\".format(usage_ratio))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65469\n"
     ]
    }
   ],
   "source": [
    "# Need to use 300 for embedding dimensions to match CN's vectors.\n",
    "embedding_dim = 300\n",
    "nb_words = len(vocab_to_int)\n",
    "\n",
    "# Create matrix with default values of zero\n",
    "word_embedding_matrix = np.zeros((nb_words, embedding_dim), dtype=np.float32)\n",
    "for word, i in vocab_to_int.items():\n",
    "    if word in embeddings_index:\n",
    "        word_embedding_matrix[i] = embeddings_index[word]\n",
    "    else:\n",
    "        # If word not in CN, create a random embedding for it\n",
    "        new_embedding = np.array(np.random.uniform(-1.0, 1.0, embedding_dim))\n",
    "        embeddings_index[word] = new_embedding\n",
    "        word_embedding_matrix[i] = new_embedding\n",
    "\n",
    "# Check if value matches len(vocab_to_int)\n",
    "print(len(word_embedding_matrix))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def convert_to_ints(text, word_count, unk_count, eos=False):\n",
    "    '''Convert words in text to an integer.\n",
    "       If word is not in vocab_to_int, use UNK's integer.\n",
    "       Total the number of words and UNKs.\n",
    "       Add EOS token to the end of texts'''\n",
    "    ints = []\n",
    "    for sentence in text:\n",
    "        sentence_ints = []\n",
    "        for word in sentence.split():\n",
    "            word_count += 1\n",
    "            if word in vocab_to_int:\n",
    "                sentence_ints.append(vocab_to_int[word])\n",
    "            else:\n",
    "                sentence_ints.append(vocab_to_int[\"<UNK>\"])\n",
    "                unk_count += 1\n",
    "        if eos:\n",
    "            sentence_ints.append(vocab_to_int[\"<EOS>\"])\n",
    "        ints.append(sentence_ints)\n",
    "    return ints, word_count, unk_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of words in headlines: 25679946\n",
      "Total number of UNKs in headlines: 170450\n",
      "Percent of words that are UNK: 0.66%\n"
     ]
    }
   ],
   "source": [
    "# Apply convert_to_ints to clean_summaries and clean_texts\n",
    "word_count = 0\n",
    "unk_count = 0\n",
    "\n",
    "int_summaries, word_count, unk_count = convert_to_ints(clean_summaries, word_count, unk_count)\n",
    "int_texts, word_count, unk_count = convert_to_ints(clean_texts, word_count, unk_count, eos=True)\n",
    "\n",
    "unk_percent = round(unk_count/word_count,4)*100\n",
    "\n",
    "print(\"Total number of words in headlines:\", word_count)\n",
    "print(\"Total number of UNKs in headlines:\", unk_count)\n",
    "print(\"Percent of words that are UNK: {}%\".format(unk_percent))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_lengths(text):\n",
    "    '''Create a data frame of the sentence lengths from a text'''\n",
    "    lengths = []\n",
    "    for sentence in text:\n",
    "        lengths.append(len(sentence))\n",
    "    return pd.DataFrame(lengths, columns=['counts'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summaries:\n",
      "              counts\n",
      "count  568412.000000\n",
      "mean        4.181620\n",
      "std         2.657872\n",
      "min         0.000000\n",
      "25%         2.000000\n",
      "50%         4.000000\n",
      "75%         5.000000\n",
      "max        48.000000\n",
      "\n",
      "Texts:\n",
      "              counts\n",
      "count  568412.000000\n",
      "mean       41.996782\n",
      "std        42.520854\n",
      "min         1.000000\n",
      "25%        18.000000\n",
      "50%        29.000000\n",
      "75%        50.000000\n",
      "max      2085.000000\n"
     ]
    }
   ],
   "source": [
    "lengths_summaries = create_lengths(int_summaries)\n",
    "lengths_texts = create_lengths(int_texts)\n",
    "\n",
    "print(\"Summaries:\")\n",
    "print(lengths_summaries.describe())\n",
    "print()\n",
    "print(\"Texts:\")\n",
    "print(lengths_texts.describe())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "84.0\n",
      "115.0\n",
      "207.0\n"
     ]
    }
   ],
   "source": [
    "# Inspect the length of texts\n",
    "print(np.percentile(lengths_texts.counts, 90))\n",
    "print(np.percentile(lengths_texts.counts, 95))\n",
    "print(np.percentile(lengths_texts.counts, 99))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8.0\n",
      "9.0\n",
      "13.0\n"
     ]
    }
   ],
   "source": [
    "# Inspect the length of summaries\n",
    "print(np.percentile(lengths_summaries.counts, 90))\n",
    "print(np.percentile(lengths_summaries.counts, 95))\n",
    "print(np.percentile(lengths_summaries.counts, 99))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def unk_counter(sentence):\n",
    "    '''Counts the number of time UNK appears in a sentence.'''\n",
    "    unk_count = 0\n",
    "    for word in sentence:\n",
    "        if word == vocab_to_int[\"<UNK>\"]:\n",
    "            unk_count += 1\n",
    "    return unk_count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "429210\n",
      "429210\n"
     ]
    }
   ],
   "source": [
    "# Sort the summaries and texts by the length of the texts, shortest to longest\n",
    "# Limit the length of summaries and texts based on the min and max ranges.\n",
    "# Remove reviews that include too many UNKs\n",
    "\n",
    "sorted_summaries = []\n",
    "sorted_texts = []\n",
    "max_text_length = 84\n",
    "max_summary_length = 13\n",
    "min_length = 2\n",
    "unk_text_limit = 1\n",
    "unk_summary_limit = 0\n",
    "\n",
    "for length in range(min(lengths_texts.counts), max_text_length): \n",
    "    for count, words in enumerate(int_summaries):\n",
    "        if (len(int_summaries[count]) >= min_length and\n",
    "            len(int_summaries[count]) <= max_summary_length and\n",
    "            len(int_texts[count]) >= min_length and\n",
    "            unk_counter(int_summaries[count]) <= unk_summary_limit and\n",
    "            unk_counter(int_texts[count]) <= unk_text_limit and\n",
    "            length == len(int_texts[count])\n",
    "           ):\n",
    "            sorted_summaries.append(int_summaries[count])\n",
    "            sorted_texts.append(int_texts[count])\n",
    "        \n",
    "# Compare lengths to ensure they match\n",
    "print(len(sorted_summaries))\n",
    "print(len(sorted_texts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Model\n",
    "\n",
    "<img src=\"f1.png\" alt=\"FAO\" width=\"490\">\n",
    "\n",
    "Bidirectional RNNs(双向网络)的改进之处便是，假设当前的输出(第t步的输出)不仅仅与前面的序列有关，并且还与后面的序列有关。\n",
    "\n",
    "例如：预测一个语句中缺失的词语那么就需要根据上下文来进行预测。Bidirectional RNNs是一个相对较简单的RNNs，是由两个RNNs上下叠加在一起组成的。输出由这两个RNNs的隐藏层的状态决定的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def model_inputs():\n",
    "    '''Create palceholders for inputs to the model'''\n",
    "    \n",
    "    input_data = tf.placeholder(tf.int32, [None, None], name='input')\n",
    "    targets = tf.placeholder(tf.int32, [None, None], name='targets')\n",
    "    lr = tf.placeholder(tf.float32, name='learning_rate')\n",
    "    keep_prob = tf.placeholder(tf.float32, name='keep_prob')\n",
    "    summary_length = tf.placeholder(tf.int32, (None,), name='summary_length')\n",
    "    max_summary_length = tf.reduce_max(summary_length, name='max_dec_len')\n",
    "    text_length = tf.placeholder(tf.int32, (None,), name='text_length')\n",
    "\n",
    "    return input_data, targets, lr, keep_prob, summary_length, max_summary_length, text_length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def process_encoding_input(target_data, vocab_to_int, batch_size):\n",
    "    '''Remove the last word id from each batch and concat the <GO> to the begining of each batch'''\n",
    "    \n",
    "    ending = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])\n",
    "    dec_input = tf.concat([tf.fill([batch_size, 1], vocab_to_int['<GO>']), ending], 1)\n",
    "\n",
    "    return dec_input"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def encoding_layer(rnn_size, sequence_length, num_layers, rnn_inputs, keep_prob):\n",
    "    '''Create the encoding layer'''\n",
    "    \n",
    "    for layer in range(num_layers):\n",
    "        with tf.variable_scope('encoder_{}'.format(layer)):\n",
    "            cell_fw = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "            cell_fw = tf.contrib.rnn.DropoutWrapper(cell_fw, \n",
    "                                                    input_keep_prob = keep_prob)\n",
    "\n",
    "            cell_bw = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                              initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "            cell_bw = tf.contrib.rnn.DropoutWrapper(cell_bw, \n",
    "                                                    input_keep_prob = keep_prob)\n",
    "\n",
    "            enc_output, enc_state = tf.nn.bidirectional_dynamic_rnn(cell_fw, \n",
    "                                                                    cell_bw, \n",
    "                                                                    rnn_inputs,\n",
    "                                                                    sequence_length,\n",
    "                                                                    dtype=tf.float32)\n",
    "    # Join outputs since we are using a bidirectional RNN\n",
    "    enc_output = tf.concat(enc_output,2)\n",
    "    \n",
    "    return enc_output, enc_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def training_decoding_layer(dec_embed_input, summary_length, dec_cell, initial_state, output_layer, \n",
    "                            vocab_size, max_summary_length):\n",
    "    '''Create the training logits'''\n",
    "    \n",
    "    training_helper = tf.contrib.seq2seq.TrainingHelper(inputs=dec_embed_input,\n",
    "                                                        sequence_length=summary_length,\n",
    "                                                        time_major=False)\n",
    "\n",
    "    training_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,\n",
    "                                                       training_helper,\n",
    "                                                       initial_state,\n",
    "                                                       output_layer) \n",
    "\n",
    "    training_logits, _ = tf.contrib.seq2seq.dynamic_decode(training_decoder,\n",
    "                                                           output_time_major=False,\n",
    "                                                           impute_finished=True,\n",
    "                                                           maximum_iterations=max_summary_length)\n",
    "    return training_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def inference_decoding_layer(embeddings, start_token, end_token, dec_cell, initial_state, output_layer,\n",
    "                             max_summary_length, batch_size):\n",
    "    '''Create the inference logits'''\n",
    "    \n",
    "    start_tokens = tf.tile(tf.constant([start_token], dtype=tf.int32), [batch_size], name='start_tokens')\n",
    "    \n",
    "    inference_helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(embeddings,\n",
    "                                                                start_tokens,\n",
    "                                                                end_token)\n",
    "                \n",
    "    inference_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,\n",
    "                                                        inference_helper,\n",
    "                                                        initial_state,\n",
    "                                                        output_layer)\n",
    "                \n",
    "    inference_logits, _ = tf.contrib.seq2seq.dynamic_decode(inference_decoder,\n",
    "                                                            output_time_major=False,\n",
    "                                                            impute_finished=True,\n",
    "                                                            maximum_iterations=max_summary_length)\n",
    "    \n",
    "    return inference_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def decoding_layer(dec_embed_input, embeddings, enc_output, enc_state, vocab_size, text_length, summary_length, \n",
    "                   max_summary_length, rnn_size, vocab_to_int, keep_prob, batch_size, num_layers):\n",
    "    '''Create the decoding cell and attention for the training and inference decoding layers'''\n",
    "    \n",
    "    for layer in range(num_layers):\n",
    "        with tf.variable_scope('decoder_{}'.format(layer)):\n",
    "            lstm = tf.contrib.rnn.LSTMCell(rnn_size,\n",
    "                                           initializer=tf.random_uniform_initializer(-0.1, 0.1, seed=2))\n",
    "            dec_cell = tf.contrib.rnn.DropoutWrapper(lstm, \n",
    "                                                     input_keep_prob = keep_prob)\n",
    "    \n",
    "    output_layer = Dense(vocab_size,\n",
    "                         kernel_initializer = tf.truncated_normal_initializer(mean = 0.0, stddev=0.1))\n",
    "    \n",
    "    attn_mech = tf.contrib.seq2seq.BahdanauAttention(rnn_size,\n",
    "                                                  enc_output,\n",
    "                                                  text_length,\n",
    "                                                  normalize=False,\n",
    "                                                  name='BahdanauAttention')\n",
    "\n",
    "    dec_cell = tf.contrib.seq2seq.DynamicAttentionWrapper(dec_cell,\n",
    "                                                          attn_mech,\n",
    "                                                          rnn_size)\n",
    "            \n",
    "    initial_state = tf.contrib.seq2seq.DynamicAttentionWrapperState(enc_state[0],\n",
    "                                                                    _zero_state_tensors(rnn_size, \n",
    "                                                                                        batch_size, \n",
    "                                                                                        tf.float32)) \n",
    "    with tf.variable_scope(\"decode\"):\n",
    "        training_logits = training_decoding_layer(dec_embed_input, \n",
    "                                                  summary_length, \n",
    "                                                  dec_cell, \n",
    "                                                  initial_state,\n",
    "                                                  output_layer,\n",
    "                                                  vocab_size, \n",
    "                                                  max_summary_length)\n",
    "    with tf.variable_scope(\"decode\", reuse=True):\n",
    "        inference_logits = inference_decoding_layer(embeddings,  \n",
    "                                                    vocab_to_int['<GO>'], \n",
    "                                                    vocab_to_int['<EOS>'],\n",
    "                                                    dec_cell, \n",
    "                                                    initial_state, \n",
    "                                                    output_layer,\n",
    "                                                    max_summary_length,\n",
    "                                                    batch_size)\n",
    "\n",
    "    return training_logits, inference_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def seq2seq_model(input_data, target_data, keep_prob, text_length, summary_length, max_summary_length, \n",
    "                  vocab_size, rnn_size, num_layers, vocab_to_int, batch_size):\n",
    "    '''Use the previous functions to create the training and inference logits'''\n",
    "    \n",
    "    # Use Numberbatch's embeddings and the newly created ones as our embeddings\n",
    "    embeddings = word_embedding_matrix\n",
    "    \n",
    "    enc_embed_input = tf.nn.embedding_lookup(embeddings, input_data)\n",
    "    enc_output, enc_state = encoding_layer(rnn_size, text_length, num_layers, enc_embed_input, keep_prob)\n",
    "    \n",
    "    dec_input = process_encoding_input(target_data, vocab_to_int, batch_size)\n",
    "    dec_embed_input = tf.nn.embedding_lookup(embeddings, dec_input)\n",
    "    \n",
    "    training_logits, inference_logits  = decoding_layer(dec_embed_input, \n",
    "                                                        embeddings,\n",
    "                                                        enc_output,\n",
    "                                                        enc_state, \n",
    "                                                        vocab_size, \n",
    "                                                        text_length, \n",
    "                                                        summary_length, \n",
    "                                                        max_summary_length,\n",
    "                                                        rnn_size, \n",
    "                                                        vocab_to_int, \n",
    "                                                        keep_prob, \n",
    "                                                        batch_size,\n",
    "                                                        num_layers)\n",
    "    \n",
    "    return training_logits, inference_logits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def pad_sentence_batch(sentence_batch):\n",
    "    \"\"\"Pad sentences with <PAD> so that each sentence of a batch has the same length\"\"\"\n",
    "    max_sentence = max([len(sentence) for sentence in sentence_batch])\n",
    "    return [sentence + [vocab_to_int['<PAD>']] * (max_sentence - len(sentence)) for sentence in sentence_batch]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_batches(summaries, texts, batch_size):\n",
    "    \"\"\"Batch summaries, texts, and the lengths of their sentences together\"\"\"\n",
    "    for batch_i in range(0, len(texts)//batch_size):\n",
    "        start_i = batch_i * batch_size\n",
    "        summaries_batch = summaries[start_i:start_i + batch_size]\n",
    "        texts_batch = texts[start_i:start_i + batch_size]\n",
    "        pad_summaries_batch = np.array(pad_sentence_batch(summaries_batch))\n",
    "        pad_texts_batch = np.array(pad_sentence_batch(texts_batch))\n",
    "        \n",
    "        # Need the lengths for the _lengths parameters\n",
    "        pad_summaries_lengths = []\n",
    "        for summary in pad_summaries_batch:\n",
    "            pad_summaries_lengths.append(len(summary))\n",
    "        \n",
    "        pad_texts_lengths = []\n",
    "        for text in pad_texts_batch:\n",
    "            pad_texts_lengths.append(len(text))\n",
    "        \n",
    "        yield pad_summaries_batch, pad_texts_batch, pad_summaries_lengths, pad_texts_lengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Set the Hyperparameters\n",
    "epochs = 100\n",
    "batch_size = 64\n",
    "rnn_size = 256\n",
    "num_layers = 2\n",
    "learning_rate = 0.005\n",
    "keep_probability = 0.75"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Graph is built.\n"
     ]
    }
   ],
   "source": [
    "# Build the graph\n",
    "train_graph = tf.Graph()\n",
    "# Set the graph to default to ensure that it is ready for training\n",
    "with train_graph.as_default():\n",
    "    \n",
    "    # Load the model inputs    \n",
    "    input_data, targets, lr, keep_prob, summary_length, max_summary_length, text_length = model_inputs()\n",
    "\n",
    "    # Create the training and inference logits\n",
    "    training_logits, inference_logits = seq2seq_model(tf.reverse(input_data, [-1]),\n",
    "                                                      targets, \n",
    "                                                      keep_prob,   \n",
    "                                                      text_length,\n",
    "                                                      summary_length,\n",
    "                                                      max_summary_length,\n",
    "                                                      len(vocab_to_int)+1,\n",
    "                                                      rnn_size, \n",
    "                                                      num_layers, \n",
    "                                                      vocab_to_int,\n",
    "                                                      batch_size)\n",
    "    \n",
    "    # Create tensors for the training logits and inference logits\n",
    "    training_logits = tf.identity(training_logits.rnn_output, 'logits')\n",
    "    inference_logits = tf.identity(inference_logits.sample_id, name='predictions')\n",
    "    \n",
    "    # Create the weights for sequence_loss\n",
    "    masks = tf.sequence_mask(summary_length, max_summary_length, dtype=tf.float32, name='masks')\n",
    "\n",
    "    with tf.name_scope(\"optimization\"):\n",
    "        # Loss function\n",
    "        cost = tf.contrib.seq2seq.sequence_loss(\n",
    "            training_logits,\n",
    "            targets,\n",
    "            masks)\n",
    "\n",
    "        # Optimizer\n",
    "        optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "\n",
    "        # Gradient Clipping\n",
    "        gradients = optimizer.compute_gradients(cost)\n",
    "        capped_gradients = [(tf.clip_by_value(grad, -5., 5.), var) for grad, var in gradients if grad is not None]\n",
    "        train_op = optimizer.apply_gradients(capped_gradients)\n",
    "print(\"Graph is built.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 234,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The shortest text length: 25\n",
      "The longest text length: 31\n"
     ]
    }
   ],
   "source": [
    "# Subset the data for training\n",
    "start = 200000\n",
    "end = start + 50000\n",
    "sorted_summaries_short = sorted_summaries[start:end]\n",
    "sorted_texts_short = sorted_texts[start:end]\n",
    "print(\"The shortest text length:\", len(sorted_texts_short[0]))\n",
    "print(\"The longest text length:\",len(sorted_texts_short[-1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   1/100 Batch   20/781 - Loss:  4.470, Seconds: 156.00\n",
      "Epoch   1/100 Batch   40/781 - Loss:  2.863, Seconds: 105.20\n",
      "Epoch   1/100 Batch   60/781 - Loss:  2.652, Seconds: 151.58\n",
      "Epoch   1/100 Batch   80/781 - Loss:  2.736, Seconds: 117.19\n",
      "Epoch   1/100 Batch  100/781 - Loss:  2.686, Seconds: 118.42\n",
      "Epoch   1/100 Batch  120/781 - Loss:  2.423, Seconds: 140.21\n",
      "Epoch   1/100 Batch  140/781 - Loss:  2.696, Seconds: 152.89\n",
      "Epoch   1/100 Batch  160/781 - Loss:  2.606, Seconds: 128.19\n",
      "Epoch   1/100 Batch  180/781 - Loss:  2.525, Seconds: 151.52\n",
      "Epoch   1/100 Batch  200/781 - Loss:  2.597, Seconds: 140.84\n",
      "Epoch   1/100 Batch  220/781 - Loss:  2.515, Seconds: 130.87\n",
      "Epoch   1/100 Batch  240/781 - Loss:  2.402, Seconds: 131.02\n",
      "Average loss for this update: 2.734\n",
      "New Record!\n",
      "Epoch   1/100 Batch  260/781 - Loss:  2.382, Seconds: 106.18\n",
      "Epoch   1/100 Batch  280/781 - Loss:  2.354, Seconds: 124.90\n",
      "Epoch   1/100 Batch  300/781 - Loss:  2.306, Seconds: 148.73\n",
      "Epoch   1/100 Batch  320/781 - Loss:  2.637, Seconds: 142.09\n",
      "Epoch   1/100 Batch  340/781 - Loss:  2.680, Seconds: 140.91\n",
      "Epoch   1/100 Batch  360/781 - Loss:  2.559, Seconds: 96.81\n",
      "Epoch   1/100 Batch  380/781 - Loss:  2.448, Seconds: 130.00\n",
      "Epoch   1/100 Batch  400/781 - Loss:  2.615, Seconds: 108.58\n",
      "Epoch   1/100 Batch  420/781 - Loss:  2.193, Seconds: 124.55\n",
      "Epoch   1/100 Batch  440/781 - Loss:  2.315, Seconds: 131.78\n",
      "Epoch   1/100 Batch  460/781 - Loss:  2.276, Seconds: 131.26\n",
      "Epoch   1/100 Batch  480/781 - Loss:  2.391, Seconds: 88.43\n",
      "Epoch   1/100 Batch  500/781 - Loss:  2.455, Seconds: 120.87\n",
      "Average loss for this update: 2.436\n",
      "New Record!\n",
      "Epoch   1/100 Batch  520/781 - Loss:  2.459, Seconds: 94.69\n",
      "Epoch   1/100 Batch  540/781 - Loss:  2.412, Seconds: 108.36\n",
      "Epoch   1/100 Batch  560/781 - Loss:  2.320, Seconds: 128.48\n",
      "Epoch   1/100 Batch  580/781 - Loss:  2.195, Seconds: 121.47\n",
      "Epoch   1/100 Batch  600/781 - Loss:  2.353, Seconds: 120.64\n",
      "Epoch   1/100 Batch  620/781 - Loss:  2.079, Seconds: 153.50\n",
      "Epoch   1/100 Batch  640/781 - Loss:  2.254, Seconds: 154.52\n",
      "Epoch   1/100 Batch  660/781 - Loss:  2.570, Seconds: 124.24\n",
      "Epoch   1/100 Batch  680/781 - Loss:  2.352, Seconds: 154.14\n",
      "Epoch   1/100 Batch  700/781 - Loss:  2.170, Seconds: 121.13\n",
      "Epoch   1/100 Batch  720/781 - Loss:  2.159, Seconds: 121.35\n",
      "Epoch   1/100 Batch  740/781 - Loss:  2.201, Seconds: 121.36\n",
      "Epoch   1/100 Batch  760/781 - Loss:  2.037, Seconds: 122.65\n",
      "Average loss for this update: 2.253\n",
      "New Record!\n",
      "Epoch   1/100 Batch  780/781 - Loss:  2.088, Seconds: 136.48\n",
      "Epoch   2/100 Batch   20/781 - Loss:  2.298, Seconds: 152.02\n",
      "Epoch   2/100 Batch   40/781 - Loss:  2.171, Seconds: 108.29\n",
      "Epoch   2/100 Batch   60/781 - Loss:  2.015, Seconds: 152.19\n",
      "Epoch   2/100 Batch   80/781 - Loss:  2.052, Seconds: 117.14\n",
      "Epoch   2/100 Batch  100/781 - Loss:  2.069, Seconds: 119.78\n",
      "Epoch   2/100 Batch  120/781 - Loss:  1.831, Seconds: 145.87\n",
      "Epoch   2/100 Batch  140/781 - Loss:  2.136, Seconds: 151.15\n",
      "Epoch   2/100 Batch  160/781 - Loss:  2.082, Seconds: 132.47\n",
      "Epoch   2/100 Batch  180/781 - Loss:  2.019, Seconds: 153.08\n",
      "Epoch   2/100 Batch  200/781 - Loss:  2.052, Seconds: 142.47\n",
      "Epoch   2/100 Batch  220/781 - Loss:  1.969, Seconds: 130.85\n",
      "Epoch   2/100 Batch  240/781 - Loss:  1.819, Seconds: 129.27\n",
      "Average loss for this update: 2.028\n",
      "New Record!\n",
      "Epoch   2/100 Batch  260/781 - Loss:  1.869, Seconds: 103.74\n",
      "Epoch   2/100 Batch  280/781 - Loss:  1.829, Seconds: 130.26\n",
      "Epoch   2/100 Batch  300/781 - Loss:  1.803, Seconds: 151.96\n",
      "Epoch   2/100 Batch  320/781 - Loss:  2.139, Seconds: 141.95\n",
      "Epoch   2/100 Batch  340/781 - Loss:  2.189, Seconds: 145.17\n",
      "Epoch   2/100 Batch  360/781 - Loss:  2.047, Seconds: 98.12\n",
      "Epoch   2/100 Batch  380/781 - Loss:  1.955, Seconds: 130.35\n",
      "Epoch   2/100 Batch  400/781 - Loss:  2.080, Seconds: 108.95\n",
      "Epoch   2/100 Batch  420/781 - Loss:  1.717, Seconds: 120.29\n",
      "Epoch   2/100 Batch  440/781 - Loss:  1.842, Seconds: 130.07\n",
      "Epoch   2/100 Batch  460/781 - Loss:  1.798, Seconds: 132.52\n",
      "Epoch   2/100 Batch  480/781 - Loss:  1.929, Seconds: 87.39\n",
      "Epoch   2/100 Batch  500/781 - Loss:  2.050, Seconds: 122.04\n",
      "Average loss for this update: 1.953\n",
      "New Record!\n",
      "Epoch   2/100 Batch  520/781 - Loss:  2.037, Seconds: 94.28\n",
      "Epoch   2/100 Batch  540/781 - Loss:  1.972, Seconds: 109.17\n",
      "Epoch   2/100 Batch  560/781 - Loss:  1.923, Seconds: 131.12\n",
      "Epoch   2/100 Batch  580/781 - Loss:  1.728, Seconds: 120.70\n",
      "Epoch   2/100 Batch  600/781 - Loss:  1.894, Seconds: 120.67\n",
      "Epoch   2/100 Batch  620/781 - Loss:  1.708, Seconds: 154.05\n",
      "Epoch   2/100 Batch  640/781 - Loss:  1.859, Seconds: 153.93\n",
      "Epoch   2/100 Batch  660/781 - Loss:  2.167, Seconds: 121.69\n",
      "Epoch   2/100 Batch  680/781 - Loss:  1.982, Seconds: 154.94\n",
      "Epoch   2/100 Batch  700/781 - Loss:  1.804, Seconds: 122.34\n",
      "Epoch   2/100 Batch  720/781 - Loss:  1.800, Seconds: 120.82\n",
      "Epoch   2/100 Batch  740/781 - Loss:  1.778, Seconds: 121.45\n",
      "Epoch   2/100 Batch  760/781 - Loss:  1.712, Seconds: 121.23\n",
      "Average loss for this update: 1.857\n",
      "New Record!\n",
      "Epoch   2/100 Batch  780/781 - Loss:  1.737, Seconds: 136.31\n",
      "Epoch   3/100 Batch   20/781 - Loss:  2.015, Seconds: 150.41\n",
      "Epoch   3/100 Batch   40/781 - Loss:  1.906, Seconds: 108.50\n",
      "Epoch   3/100 Batch   60/781 - Loss:  1.721, Seconds: 155.11\n",
      "Epoch   3/100 Batch   80/781 - Loss:  1.728, Seconds: 117.11\n",
      "Epoch   3/100 Batch  100/781 - Loss:  1.776, Seconds: 121.61\n",
      "Epoch   3/100 Batch  120/781 - Loss:  1.569, Seconds: 141.59\n",
      "Epoch   3/100 Batch  140/781 - Loss:  1.845, Seconds: 152.36\n",
      "Epoch   3/100 Batch  160/781 - Loss:  1.811, Seconds: 129.96\n",
      "Epoch   3/100 Batch  180/781 - Loss:  1.749, Seconds: 153.36\n",
      "Epoch   3/100 Batch  200/781 - Loss:  1.770, Seconds: 139.69\n",
      "Epoch   3/100 Batch  220/781 - Loss:  1.698, Seconds: 133.17\n",
      "Epoch   3/100 Batch  240/781 - Loss:  1.535, Seconds: 130.97\n",
      "Average loss for this update: 1.746\n",
      "New Record!\n",
      "Epoch   3/100 Batch  260/781 - Loss:  1.594, Seconds: 103.10\n",
      "Epoch   3/100 Batch  280/781 - Loss:  1.549, Seconds: 130.66\n",
      "Epoch   3/100 Batch  300/781 - Loss:  1.524, Seconds: 151.88\n",
      "Epoch   3/100 Batch  320/781 - Loss:  1.861, Seconds: 142.28\n",
      "Epoch   3/100 Batch  340/781 - Loss:  1.893, Seconds: 139.69\n",
      "Epoch   3/100 Batch  360/781 - Loss:  1.769, Seconds: 98.39\n",
      "Epoch   3/100 Batch  380/781 - Loss:  1.689, Seconds: 131.38\n",
      "Epoch   3/100 Batch  400/781 - Loss:  1.803, Seconds: 109.51\n",
      "Epoch   3/100 Batch  420/781 - Loss:  1.455, Seconds: 122.05\n",
      "Epoch   3/100 Batch  440/781 - Loss:  1.588, Seconds: 132.11\n",
      "Epoch   3/100 Batch  460/781 - Loss:  1.540, Seconds: 130.48\n",
      "Epoch   3/100 Batch  480/781 - Loss:  1.687, Seconds: 88.24\n",
      "Epoch   3/100 Batch  500/781 - Loss:  1.818, Seconds: 121.00\n",
      "Average loss for this update: 1.688\n",
      "New Record!\n",
      "Epoch   3/100 Batch  520/781 - Loss:  1.787, Seconds: 94.69\n",
      "Epoch   3/100 Batch  540/781 - Loss:  1.689, Seconds: 109.86\n",
      "Epoch   3/100 Batch  560/781 - Loss:  1.678, Seconds: 131.70\n",
      "Epoch   3/100 Batch  580/781 - Loss:  1.488, Seconds: 120.36\n",
      "Epoch   3/100 Batch  600/781 - Loss:  1.637, Seconds: 122.60\n",
      "Epoch   3/100 Batch  620/781 - Loss:  1.493, Seconds: 153.82\n",
      "Epoch   3/100 Batch  640/781 - Loss:  1.627, Seconds: 158.39\n",
      "Epoch   3/100 Batch  660/781 - Loss:  1.910, Seconds: 121.86\n",
      "Epoch   3/100 Batch  680/781 - Loss:  1.755, Seconds: 153.93\n",
      "Epoch   3/100 Batch  700/781 - Loss:  1.590, Seconds: 121.65\n",
      "Epoch   3/100 Batch  720/781 - Loss:  1.585, Seconds: 122.38\n",
      "Epoch   3/100 Batch  740/781 - Loss:  1.540, Seconds: 122.20\n",
      "Epoch   3/100 Batch  760/781 - Loss:  1.515, Seconds: 122.62\n",
      "Average loss for this update: 1.623\n",
      "New Record!\n",
      "Epoch   3/100 Batch  780/781 - Loss:  1.515, Seconds: 135.59\n",
      "Epoch   4/100 Batch   20/781 - Loss:  1.793, Seconds: 152.96\n",
      "Epoch   4/100 Batch   40/781 - Loss:  1.713, Seconds: 107.82\n",
      "Epoch   4/100 Batch   60/781 - Loss:  1.527, Seconds: 153.71\n",
      "Epoch   4/100 Batch   80/781 - Loss:  1.513, Seconds: 118.78\n",
      "Epoch   4/100 Batch  100/781 - Loss:  1.577, Seconds: 121.06\n",
      "Epoch   4/100 Batch  120/781 - Loss:  1.397, Seconds: 143.41\n",
      "Epoch   4/100 Batch  140/781 - Loss:  1.635, Seconds: 152.24\n",
      "Epoch   4/100 Batch  160/781 - Loss:  1.616, Seconds: 131.96\n",
      "Epoch   4/100 Batch  180/781 - Loss:  1.570, Seconds: 152.69\n",
      "Epoch   4/100 Batch  200/781 - Loss:  1.580, Seconds: 141.25\n",
      "Epoch   4/100 Batch  220/781 - Loss:  1.510, Seconds: 131.03\n",
      "Epoch   4/100 Batch  240/781 - Loss:  1.337, Seconds: 129.62\n",
      "Average loss for this update: 1.551\n",
      "New Record!\n",
      "Epoch   4/100 Batch  260/781 - Loss:  1.416, Seconds: 104.76\n",
      "Epoch   4/100 Batch  280/781 - Loss:  1.371, Seconds: 130.99\n",
      "Epoch   4/100 Batch  300/781 - Loss:  1.353, Seconds: 150.54\n",
      "Epoch   4/100 Batch  320/781 - Loss:  1.648, Seconds: 142.99\n",
      "Epoch   4/100 Batch  340/781 - Loss:  1.692, Seconds: 142.90\n",
      "Epoch   4/100 Batch  360/781 - Loss:  1.584, Seconds: 98.49\n",
      "Epoch   4/100 Batch  380/781 - Loss:  1.505, Seconds: 136.00\n",
      "Epoch   4/100 Batch  400/781 - Loss:  1.604, Seconds: 110.08\n",
      "Epoch   4/100 Batch  420/781 - Loss:  1.286, Seconds: 121.17\n",
      "Epoch   4/100 Batch  440/781 - Loss:  1.411, Seconds: 130.35\n",
      "Epoch   4/100 Batch  460/781 - Loss:  1.384, Seconds: 131.63\n",
      "Epoch   4/100 Batch  480/781 - Loss:  1.495, Seconds: 86.78\n",
      "Epoch   4/100 Batch  500/781 - Loss:  1.641, Seconds: 120.57\n",
      "Average loss for this update: 1.505\n",
      "New Record!\n",
      "Epoch   4/100 Batch  520/781 - Loss:  1.616, Seconds: 94.21\n",
      "Epoch   4/100 Batch  540/781 - Loss:  1.508, Seconds: 109.50\n",
      "Epoch   4/100 Batch  560/781 - Loss:  1.489, Seconds: 131.04\n",
      "Epoch   4/100 Batch  580/781 - Loss:  1.307, Seconds: 122.48\n",
      "Epoch   4/100 Batch  600/781 - Loss:  1.456, Seconds: 121.87\n",
      "Epoch   4/100 Batch  620/781 - Loss:  1.348, Seconds: 153.92\n",
      "Epoch   4/100 Batch  640/781 - Loss:  1.453, Seconds: 156.55\n",
      "Epoch   4/100 Batch  660/781 - Loss:  1.715, Seconds: 122.58\n",
      "Epoch   4/100 Batch  680/781 - Loss:  1.590, Seconds: 154.58\n",
      "Epoch   4/100 Batch  700/781 - Loss:  1.429, Seconds: 123.25\n",
      "Epoch   4/100 Batch  720/781 - Loss:  1.440, Seconds: 120.86\n",
      "Epoch   4/100 Batch  740/781 - Loss:  1.379, Seconds: 119.59\n",
      "Epoch   4/100 Batch  760/781 - Loss:  1.374, Seconds: 123.34\n",
      "Average loss for this update: 1.456\n",
      "New Record!\n",
      "Epoch   4/100 Batch  780/781 - Loss:  1.364, Seconds: 136.66\n",
      "Epoch   5/100 Batch   20/781 - Loss:  1.621, Seconds: 153.30\n",
      "Epoch   5/100 Batch   40/781 - Loss:  1.567, Seconds: 106.96\n",
      "Epoch   5/100 Batch   60/781 - Loss:  1.385, Seconds: 151.55\n",
      "Epoch   5/100 Batch   80/781 - Loss:  1.366, Seconds: 117.78\n",
      "Epoch   5/100 Batch  100/781 - Loss:  1.419, Seconds: 120.49\n",
      "Epoch   5/100 Batch  120/781 - Loss:  1.270, Seconds: 140.66\n",
      "Epoch   5/100 Batch  140/781 - Loss:  1.478, Seconds: 153.51\n",
      "Epoch   5/100 Batch  160/781 - Loss:  1.474, Seconds: 131.34\n",
      "Epoch   5/100 Batch  180/781 - Loss:  1.425, Seconds: 152.85\n",
      "Epoch   5/100 Batch  200/781 - Loss:  1.437, Seconds: 143.59\n",
      "Epoch   5/100 Batch  220/781 - Loss:  1.374, Seconds: 133.55\n",
      "Epoch   5/100 Batch  240/781 - Loss:  1.216, Seconds: 130.37\n",
      "Average loss for this update: 1.406\n",
      "New Record!\n",
      "Epoch   5/100 Batch  260/781 - Loss:  1.271, Seconds: 103.74\n",
      "Epoch   5/100 Batch  280/781 - Loss:  1.266, Seconds: 130.39\n",
      "Epoch   5/100 Batch  300/781 - Loss:  1.227, Seconds: 155.28\n",
      "Epoch   5/100 Batch  320/781 - Loss:  1.486, Seconds: 143.04\n",
      "Epoch   5/100 Batch  340/781 - Loss:  1.552, Seconds: 141.86\n",
      "Epoch   5/100 Batch  360/781 - Loss:  1.423, Seconds: 99.56\n",
      "Epoch   5/100 Batch  380/781 - Loss:  1.365, Seconds: 131.81\n",
      "Epoch   5/100 Batch  400/781 - Loss:  1.456, Seconds: 113.05\n",
      "Epoch   5/100 Batch  420/781 - Loss:  1.178, Seconds: 119.84\n",
      "Epoch   5/100 Batch  440/781 - Loss:  1.290, Seconds: 131.00\n",
      "Epoch   5/100 Batch  460/781 - Loss:  1.240, Seconds: 133.82\n",
      "Epoch   5/100 Batch  480/781 - Loss:  1.355, Seconds: 89.87\n",
      "Epoch   5/100 Batch  500/781 - Loss:  1.496, Seconds: 122.93\n",
      "Average loss for this update: 1.368\n",
      "New Record!\n",
      "Epoch   5/100 Batch  520/781 - Loss:  1.464, Seconds: 96.14\n",
      "Epoch   5/100 Batch  540/781 - Loss:  1.356, Seconds: 109.62\n",
      "Epoch   5/100 Batch  560/781 - Loss:  1.362, Seconds: 133.67\n",
      "Epoch   5/100 Batch  580/781 - Loss:  1.194, Seconds: 121.81\n",
      "Epoch   5/100 Batch  600/781 - Loss:  1.342, Seconds: 121.25\n",
      "Epoch   5/100 Batch  620/781 - Loss:  1.239, Seconds: 156.98\n",
      "Epoch   5/100 Batch  640/781 - Loss:  1.323, Seconds: 156.76\n",
      "Epoch   5/100 Batch  660/781 - Loss:  1.556, Seconds: 122.91\n",
      "Epoch   5/100 Batch  680/781 - Loss:  1.453, Seconds: 158.00\n",
      "Epoch   5/100 Batch  700/781 - Loss:  1.308, Seconds: 123.65\n",
      "Epoch   5/100 Batch  720/781 - Loss:  1.317, Seconds: 123.67\n",
      "Epoch   5/100 Batch  740/781 - Loss:  1.267, Seconds: 125.47\n",
      "Epoch   5/100 Batch  760/781 - Loss:  1.275, Seconds: 122.67\n",
      "Average loss for this update: 1.331\n",
      "New Record!\n",
      "Epoch   5/100 Batch  780/781 - Loss:  1.254, Seconds: 136.56\n",
      "Epoch   6/100 Batch   20/781 - Loss:  1.486, Seconds: 152.52\n",
      "Epoch   6/100 Batch   40/781 - Loss:  1.449, Seconds: 111.01\n",
      "Epoch   6/100 Batch   60/781 - Loss:  1.277, Seconds: 152.59\n",
      "Epoch   6/100 Batch   80/781 - Loss:  1.241, Seconds: 120.53\n",
      "Epoch   6/100 Batch  100/781 - Loss:  1.297, Seconds: 120.91\n",
      "Epoch   6/100 Batch  120/781 - Loss:  1.159, Seconds: 144.18\n",
      "Epoch   6/100 Batch  140/781 - Loss:  1.360, Seconds: 158.39\n",
      "Epoch   6/100 Batch  160/781 - Loss:  1.336, Seconds: 132.86\n",
      "Epoch   6/100 Batch  180/781 - Loss:  1.293, Seconds: 155.74\n",
      "Epoch   6/100 Batch  200/781 - Loss:  1.321, Seconds: 145.67\n",
      "Epoch   6/100 Batch  220/781 - Loss:  1.267, Seconds: 135.30\n",
      "Epoch   6/100 Batch  240/781 - Loss:  1.100, Seconds: 133.28\n",
      "Average loss for this update: 1.288\n",
      "New Record!\n",
      "Epoch   6/100 Batch  260/781 - Loss:  1.168, Seconds: 103.96\n",
      "Epoch   6/100 Batch  280/781 - Loss:  1.156, Seconds: 131.70\n",
      "Epoch   6/100 Batch  300/781 - Loss:  1.133, Seconds: 155.86\n",
      "Epoch   6/100 Batch  320/781 - Loss:  1.361, Seconds: 145.41\n",
      "Epoch   6/100 Batch  340/781 - Loss:  1.412, Seconds: 145.41\n",
      "Epoch   6/100 Batch  360/781 - Loss:  1.329, Seconds: 100.96\n",
      "Epoch   6/100 Batch  380/781 - Loss:  1.266, Seconds: 134.49\n",
      "Epoch   6/100 Batch  400/781 - Loss:  1.314, Seconds: 110.95\n",
      "Epoch   6/100 Batch  420/781 - Loss:  1.093, Seconds: 126.54\n",
      "Epoch   6/100 Batch  440/781 - Loss:  1.206, Seconds: 135.67\n",
      "Epoch   6/100 Batch  460/781 - Loss:  1.151, Seconds: 137.55\n",
      "Epoch   6/100 Batch  480/781 - Loss:  1.241, Seconds: 90.75\n",
      "Epoch   6/100 Batch  500/781 - Loss:  1.377, Seconds: 125.70\n",
      "Average loss for this update: 1.26\n",
      "New Record!\n",
      "Epoch   6/100 Batch  520/781 - Loss:  1.361, Seconds: 94.51\n",
      "Epoch   6/100 Batch  540/781 - Loss:  1.250, Seconds: 113.06\n",
      "Epoch   6/100 Batch  560/781 - Loss:  1.268, Seconds: 134.60\n",
      "Epoch   6/100 Batch  580/781 - Loss:  1.088, Seconds: 123.48\n",
      "Epoch   6/100 Batch  600/781 - Loss:  1.224, Seconds: 123.78\n",
      "Epoch   6/100 Batch  620/781 - Loss:  1.153, Seconds: 156.48\n",
      "Epoch   6/100 Batch  640/781 - Loss:  1.231, Seconds: 157.48\n",
      "Epoch   6/100 Batch  660/781 - Loss:  1.447, Seconds: 123.59\n",
      "Epoch   6/100 Batch  680/781 - Loss:  1.363, Seconds: 157.39\n",
      "Epoch   6/100 Batch  700/781 - Loss:  1.227, Seconds: 128.72\n",
      "Epoch   6/100 Batch  720/781 - Loss:  1.238, Seconds: 126.35\n",
      "Epoch   6/100 Batch  740/781 - Loss:  1.174, Seconds: 125.93\n",
      "Epoch   6/100 Batch  760/781 - Loss:  1.191, Seconds: 125.24\n",
      "Average loss for this update: 1.239\n",
      "New Record!\n",
      "Epoch   6/100 Batch  780/781 - Loss:  1.206, Seconds: 139.27\n",
      "Epoch   7/100 Batch   20/781 - Loss:  1.416, Seconds: 155.27\n",
      "Epoch   7/100 Batch   40/781 - Loss:  1.381, Seconds: 110.29\n",
      "Epoch   7/100 Batch   60/781 - Loss:  1.193, Seconds: 156.02\n",
      "Epoch   7/100 Batch   80/781 - Loss:  1.177, Seconds: 122.76\n",
      "Epoch   7/100 Batch  100/781 - Loss:  1.233, Seconds: 122.76\n",
      "Epoch   7/100 Batch  120/781 - Loss:  1.110, Seconds: 144.25\n",
      "Epoch   7/100 Batch  140/781 - Loss:  1.274, Seconds: 156.67\n",
      "Epoch   7/100 Batch  160/781 - Loss:  1.251, Seconds: 138.95\n",
      "Epoch   7/100 Batch  180/781 - Loss:  1.230, Seconds: 159.33\n",
      "Epoch   7/100 Batch  200/781 - Loss:  1.244, Seconds: 147.36\n",
      "Epoch   7/100 Batch  220/781 - Loss:  1.189, Seconds: 136.09\n",
      "Epoch   7/100 Batch  240/781 - Loss:  1.037, Seconds: 136.38\n",
      "Average loss for this update: 1.217\n",
      "New Record!\n",
      "Epoch   7/100 Batch  260/781 - Loss:  1.106, Seconds: 103.50\n",
      "Epoch   7/100 Batch  280/781 - Loss:  1.092, Seconds: 135.91\n",
      "Epoch   7/100 Batch  300/781 - Loss:  1.064, Seconds: 156.83\n",
      "Epoch   7/100 Batch  320/781 - Loss:  1.267, Seconds: 146.38\n",
      "Epoch   7/100 Batch  340/781 - Loss:  1.321, Seconds: 145.75\n",
      "Epoch   7/100 Batch  360/781 - Loss:  1.238, Seconds: 101.32\n",
      "Epoch   7/100 Batch  380/781 - Loss:  1.187, Seconds: 133.62\n",
      "Epoch   7/100 Batch  400/781 - Loss:  1.232, Seconds: 114.49\n",
      "Epoch   7/100 Batch  420/781 - Loss:  1.007, Seconds: 126.89\n",
      "Epoch   7/100 Batch  440/781 - Loss:  1.119, Seconds: 140.93\n",
      "Epoch   7/100 Batch  460/781 - Loss:  1.067, Seconds: 139.15\n",
      "Epoch   7/100 Batch  480/781 - Loss:  1.148, Seconds: 91.13\n",
      "Epoch   7/100 Batch  500/781 - Loss:  1.280, Seconds: 126.78\n",
      "Average loss for this update: 1.175\n",
      "New Record!\n",
      "Epoch   7/100 Batch  520/781 - Loss:  1.278, Seconds: 126.51\n",
      "Epoch   7/100 Batch  540/781 - Loss:  1.166, Seconds: 116.39\n",
      "Epoch   7/100 Batch  560/781 - Loss:  1.180, Seconds: 176.67\n",
      "Epoch   7/100 Batch  580/781 - Loss:  1.019, Seconds: 123.86\n",
      "Epoch   7/100 Batch  600/781 - Loss:  1.155, Seconds: 160.57\n",
      "Epoch   7/100 Batch  620/781 - Loss:  1.081, Seconds: 229.01\n",
      "Epoch   7/100 Batch  640/781 - Loss:  1.144, Seconds: 213.84\n",
      "Epoch   7/100 Batch  660/781 - Loss:  1.352, Seconds: 215.01\n",
      "Epoch   7/100 Batch  680/781 - Loss:  1.281, Seconds: 197.15\n",
      "Epoch   7/100 Batch  700/781 - Loss:  1.159, Seconds: 195.18\n",
      "Epoch   7/100 Batch  720/781 - Loss:  1.155, Seconds: 210.13\n",
      "Epoch   7/100 Batch  740/781 - Loss:  1.104, Seconds: 187.91\n",
      "Epoch   7/100 Batch  760/781 - Loss:  1.114, Seconds: 176.62\n",
      "Average loss for this update: 1.159\n",
      "New Record!\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-158-3fdb2639b0d7>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     29\u001b[0m                  \u001b[0msummary_length\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0msummaries_lengths\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m                  \u001b[0mtext_length\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mtexts_lengths\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 31\u001b[0;31m                  keep_prob: keep_probability})\n\u001b[0m\u001b[1;32m     32\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     33\u001b[0m             \u001b[0mbatch_loss\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0mloss\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m//anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36mrun\u001b[0;34m(self, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    776\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    777\u001b[0m       result = self._run(None, fetches, feed_dict, options_ptr,\n\u001b[0;32m--> 778\u001b[0;31m                          run_metadata_ptr)\n\u001b[0m\u001b[1;32m    779\u001b[0m       \u001b[0;32mif\u001b[0m \u001b[0mrun_metadata\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    780\u001b[0m         \u001b[0mproto_data\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtf_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mTF_GetBuffer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrun_metadata_ptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m//anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run\u001b[0;34m(self, handle, fetches, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m    980\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mfinal_fetches\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mfinal_targets\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    981\u001b[0m       results = self._do_run(handle, final_targets, final_fetches,\n\u001b[0;32m--> 982\u001b[0;31m                              feed_dict_string, options, run_metadata)\n\u001b[0m\u001b[1;32m    983\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    984\u001b[0m       \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m//anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_run\u001b[0;34m(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\u001b[0m\n\u001b[1;32m   1030\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mhandle\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1031\u001b[0m       return self._do_call(_run_fn, self._session, feed_dict, fetch_list,\n\u001b[0;32m-> 1032\u001b[0;31m                            target_list, options, run_metadata)\n\u001b[0m\u001b[1;32m   1033\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1034\u001b[0m       return self._do_call(_prun_fn, self._session, handle, feed_dict,\n",
      "\u001b[0;32m//anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_do_call\u001b[0;34m(self, fn, *args)\u001b[0m\n\u001b[1;32m   1037\u001b[0m   \u001b[0;32mdef\u001b[0m \u001b[0m_do_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1038\u001b[0m     \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1039\u001b[0;31m       \u001b[0;32mreturn\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1040\u001b[0m     \u001b[0;32mexcept\u001b[0m \u001b[0merrors\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mOpError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1041\u001b[0m       \u001b[0mmessage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcompat\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mas_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmessage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m//anaconda/lib/python3.5/site-packages/tensorflow/python/client/session.py\u001b[0m in \u001b[0;36m_run_fn\u001b[0;34m(session, feed_dict, fetch_list, target_list, options, run_metadata)\u001b[0m\n\u001b[1;32m   1019\u001b[0m         return tf_session.TF_Run(session, options,\n\u001b[1;32m   1020\u001b[0m                                  \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtarget_list\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1021\u001b[0;31m                                  status, run_metadata)\n\u001b[0m\u001b[1;32m   1022\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1023\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_prun_fn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfeed_dict\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfetch_list\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ],
     "output_type": "error"
    }
   ],
   "source": [
    "# Train the Model\n",
    "learning_rate_decay = 0.95\n",
    "min_learning_rate = 0.0005\n",
    "display_step = 20 # Check training loss after every 20 batches\n",
    "stop_early = 0 \n",
    "stop = 3 # If the update loss does not decrease in 3 consecutive update checks, stop training\n",
    "per_epoch = 3 # Make 3 update checks per epoch\n",
    "update_check = (len(sorted_texts_short)//batch_size//per_epoch)-1\n",
    "\n",
    "update_loss = 0 \n",
    "batch_loss = 0\n",
    "summary_update_loss = [] # Record the update losses for saving improvements in the model\n",
    "\n",
    "checkpoint = \"best_model.ckpt\" \n",
    "with tf.Session(graph=train_graph) as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # If we want to continue training a previous session\n",
    "    #loader = tf.train.import_meta_graph(\"./\" + checkpoint + '.meta')\n",
    "    #loader.restore(sess, checkpoint)\n",
    "    \n",
    "    for epoch_i in range(1, epochs+1):\n",
    "        update_loss = 0\n",
    "        batch_loss = 0\n",
    "        for batch_i, (summaries_batch, texts_batch, summaries_lengths, texts_lengths) in enumerate(\n",
    "                get_batches(sorted_summaries_short, sorted_texts_short, batch_size)):\n",
    "            start_time = time.time()\n",
    "            _, loss = sess.run(\n",
    "                [train_op, cost],\n",
    "                {input_data: texts_batch,\n",
    "                 targets: summaries_batch,\n",
    "                 lr: learning_rate,\n",
    "                 summary_length: summaries_lengths,\n",
    "                 text_length: texts_lengths,\n",
    "                 keep_prob: keep_probability})\n",
    "\n",
    "            batch_loss += loss\n",
    "            update_loss += loss\n",
    "            end_time = time.time()\n",
    "            batch_time = end_time - start_time\n",
    "\n",
    "            if batch_i % display_step == 0 and batch_i > 0:\n",
    "                print('Epoch {:>3}/{} Batch {:>4}/{} - Loss: {:>6.3f}, Seconds: {:>4.2f}'\n",
    "                      .format(epoch_i,\n",
    "                              epochs, \n",
    "                              batch_i, \n",
    "                              len(sorted_texts_short) // batch_size, \n",
    "                              batch_loss / display_step, \n",
    "                              batch_time*display_step))\n",
    "                batch_loss = 0\n",
    "\n",
    "            if batch_i % update_check == 0 and batch_i > 0:\n",
    "                print(\"Average loss for this update:\", round(update_loss/update_check,3))\n",
    "                summary_update_loss.append(update_loss)\n",
    "                \n",
    "                # If the update loss is at a new minimum, save the model\n",
    "                if update_loss <= min(summary_update_loss):\n",
    "                    print('New Record!') \n",
    "                    stop_early = 0\n",
    "                    saver = tf.train.Saver() \n",
    "                    saver.save(sess, checkpoint)\n",
    "\n",
    "                else:\n",
    "                    print(\"No Improvement.\")\n",
    "                    stop_early += 1\n",
    "                    if stop_early == stop:\n",
    "                        break\n",
    "                update_loss = 0\n",
    "            \n",
    "                    \n",
    "        # Reduce learning rate, but not below its minimum value\n",
    "        learning_rate *= learning_rate_decay\n",
    "        if learning_rate < min_learning_rate:\n",
    "            learning_rate = min_learning_rate\n",
    "        \n",
    "        if stop_early == stop:\n",
    "            print(\"Stopping Training.\")\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 测试效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def text_to_seq(text):\n",
    "    '''Prepare the text for the model'''\n",
    "    \n",
    "    text = clean_text(text)\n",
    "    return [vocab_to_int.get(word, vocab_to_int['<UNK>']) for word in text.split()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./best_model.ckpt\n",
      "Original Text: love individual oatmeal cups found years ago sam quit selling sound big lots quit selling found target expensive buy individually trilled get entire case time go anywhere need water microwave spoon know quaker flavor packets\n",
      "\n",
      "Text\n",
      "  Word Ids:    [70595, 18808, 668, 45565, 51927, 51759, 32488, 13510, 32036, 59599, 11693, 444, 23335, 32036, 59599, 51927, 67316, 726, 24842, 50494, 48492, 1062, 44749, 38443, 42344, 67973, 14168, 7759, 5347, 29528, 58763, 18927, 17701, 20232, 47328]\n",
      "  Input Words: love individual oatmeal cups found years ago sam quit selling sound big lots quit selling found target expensive buy individually trilled get entire case time go anywhere need water microwave spoon know quaker flavor packets\n",
      "\n",
      "Summary\n",
      "  Word Ids:       [70595, 28738]\n",
      "  Response Words: love it\n"
     ]
    }
   ],
   "source": [
    "# Create your own review or use one from the dataset\n",
    "#input_sentence = \"I have never eaten an apple before, but this red one was nice. \\\n",
    "                  #I think that I will try a green apple next time.\"\n",
    "#text = text_to_seq(input_sentence)\n",
    "random = np.random.randint(0,len(clean_texts))\n",
    "input_sentence = clean_texts[random]\n",
    "text = text_to_seq(clean_texts[random])\n",
    "\n",
    "checkpoint = \"./best_model.ckpt\"\n",
    "\n",
    "loaded_graph = tf.Graph()\n",
    "with tf.Session(graph=loaded_graph) as sess:\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(checkpoint + '.meta')\n",
    "    loader.restore(sess, checkpoint)\n",
    "\n",
    "    input_data = loaded_graph.get_tensor_by_name('input:0')\n",
    "    logits = loaded_graph.get_tensor_by_name('predictions:0')\n",
    "    text_length = loaded_graph.get_tensor_by_name('text_length:0')\n",
    "    summary_length = loaded_graph.get_tensor_by_name('summary_length:0')\n",
    "    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')\n",
    "    \n",
    "    #Multiply by batch_size to match the model's input parameters\n",
    "    answer_logits = sess.run(logits, {input_data: [text]*batch_size, \n",
    "                                      summary_length: [np.random.randint(5,8)], \n",
    "                                      text_length: [len(text)]*batch_size,\n",
    "                                      keep_prob: 1.0})[0] \n",
    "\n",
    "# Remove the padding from the tweet\n",
    "pad = vocab_to_int[\"<PAD>\"] \n",
    "\n",
    "print('Original Text:', input_sentence)\n",
    "\n",
    "print('\\nText')\n",
    "print('  Word Ids:    {}'.format([i for i in text]))\n",
    "print('  Input Words: {}'.format(\" \".join([int_to_vocab[i] for i in text])))\n",
    "\n",
    "print('\\nSummary')\n",
    "print('  Word Ids:       {}'.format([i for i in answer_logits if i != pad]))\n",
    "print('  Response Words: {}'.format(\" \".join([int_to_vocab[i] for i in answer_logits if i != pad])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Examples of reviews and summaries:\n",
    "- Review(1): The coffee tasted great and was at such a good price! I highly recommend this to everyone!\n",
    "- Summary(1): great coffee\n",
    "\n",
    "\n",
    "- Review(2): This is the worst cheese that I have ever bought! I will never buy it again and I hope you won't either!\n",
    "- Summary(2): omg gross gross\n",
    "\n",
    "\n",
    "- Review(3): love individual oatmeal cups found years ago sam quit selling sound big lots quit selling found target expensive buy individually trilled get entire case time go anywhere need water microwave spoon know quaker flavor packets\n",
    "- Summary(3): love it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
